feat: Added bash script for updating `data.yaml` #2769

Rishab87 · 2025-09-07T06:13:08Z

What this PR does / why we need it:
This PR automates the process of updating the data.yaml file. To run specify the new version like this:

./update-data-yaml.sh v2.18.0

For each new version, it fetches the latest Kubernetes releases (we keep the k8s version one behind the latest release), adds the new compatibility mapping, and prunes the oldest entry. This ensures our compatibility data is kept current automatically with each new release.

How does this change affect the cardinality of KSM: (increases, decreases or does not change cardinality)
does not change

Which issue(s) this PR fixes: (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged)
Related to #2756

k8s-ci-robot · 2025-09-07T06:13:16Z

This issue is currently awaiting triage.

If kube-state-metrics contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot · 2025-09-07T06:13:17Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Rishab87
Once this PR has been reviewed and has the lgtm label, please assign catherinef-dev for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mrueg · 2025-09-08T13:53:50Z

scripts/update-data-yaml.sh

+echo "Fetching latest Kubernetes version..."
+LATEST_K8S_FULL_VERSION=$(curl --silent --fail "https://api.github.com/repos/kubernetes/kubernetes/releases/latest" | jq -r '.tag_name')
+
+if [ -z "$LATEST_K8S_FULL_VERSION" ] || [ "$LATEST_K8S_FULL_VERSION" == "null" ]; then
+    echo "Error: Failed to fetch the latest Kubernetes version from GitHub." >&2
+    exit 1
+fi
+
+LATEST_K8S_VERSION=$(echo "${LATEST_K8S_FULL_VERSION}" | sed 's/^v//' | cut -d. -f1,2)
+echo "Latest stable Kubernetes version (N): ${LATEST_K8S_VERSION}"
+
+K8S_MAJOR=$(echo "${LATEST_K8S_VERSION}" | cut -d. -f1)
+K8S_MINOR=$(echo "${LATEST_K8S_VERSION}" | cut -d. -f2)
+PREV_K8S_MINOR=$((K8S_MINOR - 1))
+K8S_VERSION_FOR_NEW_RELEASE="${K8S_MAJOR}.${PREV_K8S_MINOR}"
+echo "New release ${NEW_VERSION_WITH_V} will be mapped to Kubernetes (N-1): ${K8S_VERSION_FOR_NEW_RELEASE}"


I think it should check k8s.io/client-go's version in go.mod when bumping, if I understand it correctly. :)

yup right, changed it

mrueg · 2025-09-08T20:52:54Z

scripts/update-data-yaml.sh

+
+check_command "yq"
+check_command "jq"
+check_command "curl"


Looks like we don't need curl anymore. :)

mrueg · 2025-09-08T20:54:49Z

scripts/update-data-yaml.sh

+check_command "yq"
+check_command "jq"


Can we use github.com/itchyny/gojq/cmd/gojq instead?

Once #2660 is in, it will be used anyway as part of the project. It supports yaml via --yaml-input --yaml-output args.

added this, havent pushed go.mod and go.sum since this is to be merged after that PR

mrueg · 2025-09-08T20:55:41Z

scripts/update-data-yaml.sh

+    exit 1
+fi
+
+CLIENT_GO_FULL_VERSION=$(grep 'k8s.io/client-go' "${GO_MOD_FILE}" | awk '{print $2}')


You can get the version like this: https://github.com/kubernetes/kube-state-metrics/pull/2739/files#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52R23

Rishab87 · 2025-09-09T14:29:22Z

@mrueg I've addressed all the reviews, can you please re-review?

rexagod

Also, generally I'd like to see the more complex and lengthy pipe-based commands here broken down and commented at each step (concisely) so they're easier to maintain.

-FOO=$(a|b|c)
+FOO=$(\
+# what "a" does
+ a|\
+# what "b" does
+ b|\
+# what "c" does
+ c\
+)

rexagod · 2025-09-24T21:23:12Z

scripts/update-data-yaml.sh

+    echo "${JSON_DATA}" | gojq -r --arg version "${NEW_VERSION_WITH_V}" --arg k8s "${K8S_VERSION_FOR_NEW_RELEASE}" '
+        .compat[] | select(.version != $version) | "  - kubernetes: \"" + .kubernetes + "\"\n    version: " + .version
+    ' >> "${DATA_FILE}"
+
+    echo "  - kubernetes: \"${K8S_VERSION_FOR_NEW_RELEASE}\"" >> "${DATA_FILE}"
+    echo "    version: ${NEW_VERSION_WITH_V}" >> "${DATA_FILE}"


I maybe off but wouldn't append main to the list as well?

yes it's being appended separately below

scripts/update-data-yaml.sh

rexagod · 2025-09-24T21:50:46Z

scripts/update-data-yaml.sh

+VERSIONS_LIST=$(echo "${JSON_DATA}" | gojq -r '.compat[] | select(.version != "main") | "\(.version)|\(.kubernetes)"' 2>/dev/null || true)
+FULL_VERSIONS_LIST=$(printf "%s\n%s|%s" "${VERSIONS_LIST}" "${NEW_VERSION_WITH_V}" "${K8S_VERSION_FOR_NEW_RELEASE}")
+
+SORTED_VERSIONS=$(echo "${FULL_VERSIONS_LIST}" | grep -v '^$' | sort -t'|' -k1,1 -Vr | head -n "${NUM_RELEASES_TO_KEEP}" | sort -t'|' -k1,1 -V)


Again, I maybe off here since I've not run this, but wouldn't this take into account the current 5 releases, instead of dropping the oldest one, and appending the given release and then main?

yes it takes into account current 5 releases excluding the main, so it removes the oldest release, adds the new one and updates the main version too if needed

scripts/update-data-yaml.sh

Rishab87 · 2025-09-25T07:43:32Z

@rexagod I've made the suggested changes, can you please re-review it? Also I've updated go.sum and go.mod to add gojq as a dependency in this PR, so this PR does not depend on #2660 to get merged

mrueg · 2025-09-23T20:10:14Z

scripts/update-data-yaml.sh

+K8S_MINOR=$(echo "${CLIENT_GO_FULL_VERSION}" | cut -d. -f2)
+
+K8S_VERSION_FOR_NEW_RELEASE="1.${K8S_MINOR}"
+echo "New release ${NEW_VERSION_WITH_V} will be mapped to Kubernetes (N-1): ${K8S_VERSION_FOR_NEW_RELEASE}"


What do you mean with the n-1 here?

added that by mistake I was thinking something while writing down this script, removing it.

I think that's supposed to mean the latest-1 release, relative to the latest KSM release.

yeah I think so I added it earlier because I was subtracting one version behind the latest k8s version but later Manuel suggested to take the k8s version from go.mod instead of fetching from github api and subtracting it.

mrueg · 2025-09-23T20:10:56Z

scripts/update-data-yaml.sh

+echo "New release ${NEW_VERSION_WITH_V} will be mapped to Kubernetes (N-1): ${K8S_VERSION_FOR_NEW_RELEASE}"
+
+
+JSON_DATA=$(cat "${DATA_FILE}" | gojq -r --yaml-input '.')


Do you need to use cat or can gojq also read from a file?

Yes, gojq can read directly from a file without using cat. I've already removed cat from here

mrueg · 2025-09-25T10:28:40Z

scripts/update-data-yaml.sh

+EXISTING_EXACT_MATCH=$(\
+echo "${JSON_DATA}" |\
+# Query for existing entry with same version and k8s mapping
+gojq -r --arg version "${NEW_VERSION_WITH_V}" --arg k8s "${K8S_VERSION_FOR_NEW_RELEASE}" ".compat[] | select(.version == \$version and .kubernetes == \$k8s) | .version // empty"\


You can call gojq via go tool similar to how we do it in the makefile

Rishab87 · 2025-09-25T10:55:56Z

@mrueg I've addressed all the reviews, can you please re-review it?

Rishab87 · 2025-09-29T10:45:23Z

@mrueg @rexagod just a follow up on this one

mrueg · 2025-09-29T10:46:48Z

scripts/update-data-yaml.sh

@@ -0,0 +1,152 @@
+#!/bin/bash


I believe this file isn't executable by default, can you run chmod +x and commit it again?

mrueg · 2025-09-29T11:23:05Z

scripts/update-data-yaml.sh

+# Remove empty lines
+grep -v '^$' |\
+# Sort by version (descending) using version sort
+sort -t'|' -k1,1 -Vr |\


Why do we need to sort twice here? Could we use tail instead of head?

yup you're right, thanks for pointing it out made the changes

scripts/update-data-yaml.sh

rexagod · 2025-10-01T03:07:18Z

scripts/update-data-yaml.sh

+
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+REPO_ROOT=$( cd -- "${SCRIPT_DIR}/.." &> /dev/null && pwd )
+DATA_FILE="${REPO_ROOT}/data.yaml"


How about maintaining a couple of mock data files (inputs and outputs), and populating them with the various use-cases covered here, to ensure functionality doesn't break over time?

yeah nice idea, added them

scripts/update-data-yaml.sh

Rishab87 · 2025-10-01T17:57:44Z

@rexagod I've addressed all your reviews, can you please re-review it?

Rishab87 · 2025-10-07T07:48:50Z

@rexagod just a follow up on this one.

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Sep 7, 2025

k8s-ci-robot requested review from dgrisonnet and mrueg September 7, 2025 06:13

github-project-automation bot added this to SIG Instrumentation Sep 7, 2025

github-project-automation bot moved this to Needs Triage in SIG Instrumentation Sep 7, 2025

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Sep 7, 2025

Rishab87 force-pushed the data-yaml-script branch from 8c35fe7 to 92ca55e Compare September 7, 2025 06:19

mrueg reviewed Sep 8, 2025

View reviewed changes

Rishab87 force-pushed the data-yaml-script branch from 92ca55e to 97387e7 Compare September 8, 2025 14:06

Rishab87 requested a review from mrueg September 8, 2025 14:08

mrueg reviewed Sep 8, 2025

View reviewed changes

Rishab87 force-pushed the data-yaml-script branch from 97387e7 to c831c83 Compare September 9, 2025 14:28

Rishab87 requested a review from mrueg September 9, 2025 14:29

rexagod reviewed Sep 24, 2025

View reviewed changes

Rishab87 force-pushed the data-yaml-script branch from c831c83 to f564d13 Compare September 25, 2025 07:39

Rishab87 force-pushed the data-yaml-script branch from f564d13 to a098852 Compare September 25, 2025 07:48

Rishab87 requested a review from rexagod September 25, 2025 07:48

Rishab87 force-pushed the data-yaml-script branch from a098852 to 37d117a Compare September 25, 2025 09:47

mrueg reviewed Sep 25, 2025

View reviewed changes

Rishab87 force-pushed the data-yaml-script branch from 37d117a to 7f9ca0f Compare September 25, 2025 10:55

Rishab87 requested a review from mrueg September 25, 2025 10:56

mrueg reviewed Sep 29, 2025

View reviewed changes

Rishab87 force-pushed the data-yaml-script branch from 7f9ca0f to 4344ef3 Compare September 29, 2025 11:05

Rishab87 requested a review from mrueg September 29, 2025 11:05

mrueg reviewed Sep 29, 2025

View reviewed changes

Rishab87 force-pushed the data-yaml-script branch from 4344ef3 to 84f24be Compare September 29, 2025 11:34

Rishab87 requested a review from mrueg September 29, 2025 11:35

Rishab87 mentioned this pull request Sep 30, 2025

feat: Added a workflow for automating pre-release PR #2787

Open

rexagod reviewed Oct 1, 2025

View reviewed changes

added bash script to update data.yaml

a77dd36

Rishab87 force-pushed the data-yaml-script branch from 84f24be to a77dd36 Compare October 1, 2025 17:56

Rishab87 requested a review from rexagod October 1, 2025 17:57

rexagod assigned mrueg and rexagod Oct 3, 2025

rexagod moved this from Needs Triage to In Progress in SIG Instrumentation Oct 3, 2025

		echo "New release ${NEW_VERSION_WITH_V} will be mapped to Kubernetes (N-1): ${K8S_VERSION_FOR_NEW_RELEASE}"


		JSON_DATA=$(cat "${DATA_FILE}" \| gojq -r --yaml-input '.')

		check_command "yq"
		check_command "jq"

feat: Added bash script for updating data.yaml #2769

Are you sure you want to change the base?

feat: Added bash script for updating data.yaml #2769

Conversation

Rishab87 commented Sep 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Sep 7, 2025

Uh oh!

k8s-ci-robot commented Sep 7, 2025

Uh oh!

mrueg Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Rishab87 Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Rishab87 commented Sep 9, 2025

Uh oh!

rexagod left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Rishab87 commented Sep 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rexagod Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Rishab87 commented Sep 25, 2025

Uh oh!

Rishab87 commented Sep 29, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

feat: Added bash script for updating `data.yaml` #2769

feat: Added bash script for updating `data.yaml` #2769

Rishab87 commented Sep 7, 2025 •

edited

Loading

mrueg Sep 8, 2025 •

edited

Loading

Rishab87 Sep 9, 2025 •

edited

Loading

rexagod Oct 1, 2025 •

edited

Loading